As compared to conventional cathode-ray tubes (CRTs) primarily used for realizing moving images, LCDs (Liquid Crystal Displays) have a drawback, so-called motion blur, which is, blurring of outline of a motion portion perceived by a viewer when displaying an image with motion. It is suggested that this motion blur arises from the LCD display mode itself (see, e.g., patent document 1).
Since fluorescent material is scanned by an electron beam to cause emission of light for display in CRTs, the light emission of each pixel is basically impulse-like although slight afterglow of the fluorescent material exists. This is called an impulse display mode. On the other hand, in the case of LCDs, an electric charge is accumulated by applying an electric field to liquid crystal and is retained at a relatively high rate until the next time the electric field is applied. Especially, in the case of the TFT mode, since a TFT switch is disposed for each dot configuring a pixel and each pixel normally has an auxiliary capacity, the ability to retain the accumulated charge is extremely high. Therefore, the light emission is continued until the pixels are rewritten by the application of the electric field based on image information of the next frame or field (hereinafter, represented by the frame). This is called a hold display mode.
Since impulse response of image displaying light has a temporal spread in the above hold display mode, temporal frequency characteristics are deteriorated along with spatial frequency characteristics, resulting in the motion blur. That is, since the human eyes can smoothly follow a moving object, if the light emission time is long as in the case of the hold type, motion of image seems jerky and unnatural due to a time integration effect.
To improve the motion blur in the above hold display mode, a frame rate (number of frames) is converted by interpolating an image between frames in a known technology. This technology is called FRC (Frame Rate Converter) and is put to practical use in liquid crystal displaying apparatuses, etc.
Conventionally known methods of converting the frame rate include various techniques such as simply repeating read-out of the same frame for a plurality of times and frame interpolation using linear interpolation between frames. However, in the case of the frame interpolation using linear interpolation between frames, unnaturalness of motion (jerkiness, judder) is generated due to the frame rate conversion, and the motion blur disturbance due to the above hold display mode cannot sufficiently be improved, resulting in inadequate image quality.
To eliminate effects of the jerkiness, etc., and improve quality of moving images, a motion compensated frame interpolation (motion compensation) process using motion vectors has been proposed. In this motion compensation process, since a moving image itself is captured and compensated, highly natural moving images can be acquired without deteriorating the resolution and generating the jerkiness. Since interpolation image signals are generated with motion compensation, the motion blur disturbance due to the above hold display mode can sufficiently be improved.
Above patent document 1 discloses a technology of motion-adaptively generating interpolation frames to increase a frame frequency of a display image for improving deterioration of spatial frequency characteristics causing the motion blur. In this case, at least one interpolation image signal interpolated between frames of a display image is motion-adaptively created from the previous and subsequent frames, and the created interpolation image signals are interpolated between the frames and are sequentially displayed.
By increasing the display frame frequency with the motion compensated frame interpolation process using the motion vector information, the display state of LCDs (hold display mode) can be made closer to the display state of CRTs (impulse display mode), and an improvement can be made in the image quality deterioration due to the motion blur generated when displaying moving images.
In the motion compensated frame interpolation process, it is essential to detect the motion vectors for the motion compensation. For example, the block matching method, gradient method, etc., are proposed as representative techniques for the motion vector detection. In the gradient method, the motion vector is detected for each pixel or small block between two consecutive frames and this is used to interpolate each pixel or small block of the interpolation frame between two frames. That is, an image at an arbitrary position between two frames is interpolated at an accurately compensated position to convert the number of frames.
Conventionally, an interpolation processing method giving priority to zero-vector is known that the interpolation process is performed with the use of zero-vector instead of the detected vector if no difference exists between the previous frame and the current frame, that is, for a still region, (see, e.g., patent document 2, non-patent document 1). This process is effective for improving the image quality of background images. For example, according to an invention described in patent document 2, since the background is in a substantially still state as compared to the foreground when a camera remains stationary, the zero-vector preferential interpolation process is executed for the background region, and the zero-vector preferential interpolation process is not executed for the foreground region.
FIG. 24 is a block diagram of a main configuration of an interpolation frame generating process of a conventional frame rate converting portion, and in FIG. 24, 101 is a delaying portion; 102 is a still region detecting portion; 103 is an interpolation vector evaluating portion; 104 is an interpolation vector controlling portion; and 105 is an interpolation frame generating portion. An input image signal is delayed for one frame period by the delaying portion 101, and each image signal of a previous frame and a current frame is input to the interpolation vector evaluating portion 103, the still region detecting portion 102, and the interpolation frame generating portion 105.
A motion vector detecting portion not shown selects a motion vector closest to the motion vector of the detected block for an initial vector from a vector memory (not shown) having accumulated thereon motion vectors already detected in a frame right before the previous frame as initial vector candidates (estimation vectors), and the selected initial vector is used as a starting point to detect a motion vector between the previous frame and the current frame with gradient method calculations.
The interpolation vector evaluating portion 103 evaluates the motion vector detected by the motion vector detecting portion and executes a process of allocating an optimum interpolation vector to an interpolation block of an interpolation frame based on the evaluation result. That is, the interpolation vector evaluating portion 103 uses the estimation vectors sequentially input from the vector memory to calculate DFD (Displaced Field Difference) and determines and outputs the optimum interpolation vector based on this DFD.
DFD is an index indicating an accuracy degree of a motion vector (the estimation vector in this case) and is an absolute value sum of inter-frame differences between each pixel in the detected block and each pixel in a block pointed by the motion vector from the detected block. Therefore, the smaller a DFD value is, the better the matching becomes between the detected block and the block pointed by the motion vector from the detected block, which indicates that the corresponding candidate vector is more appropriate.
In a technology proposed in above non-patent document 1, a flag β is set to perform motion detection of an input image, in other words, to determine a still region. The still region means a region with a slow motion or no motion.
The still region detecting portion 102 sets the above flag β, calculates FD (Field Difference), which is DFD for zero-vector, and compares the FD with a predetermined threshold to detect the still region. In this case, the DFD is calculated using the interpolation vector output from the interpolation vector evaluating portion 103, and the minimum value Min(DFD) of the DFD is used as the predetermined threshold to output β=1 when Min(DFD)<FD and otherwise output β=0. That is, the still region detecting portion 102 outputs β=1 for the moving region and β=0 for the still region to the interpolation vector controlling portion 104 based on the inter-frame differences of the input image signals.
The above determination condition of the still region is not limited to the condition of Min(DFD)<FD, and even if the condition of Min(DFD)<FD is not satisfied, β=1 (i.e., detection of the moving region) may be output when the absolute value of Min(DFD) is sufficiently small. Alternatively, for example, such a condition as Min(DFD)×k<FD(0≦k≦1) is set to rigidify the detection criterion of the still region.
The interpolation vector controlling portion 104 outputs a control signal to the interpolation frame generating portion 105 such that the interpolation vector from the interpolation vector evaluating portion 103 is directly used to generate an interpolation frame if β=1 is input from the still region detecting portion 102 and such that zero-vector is used instead of the interpolation vector from the interpolation vector evaluating portion 103 to generate the interpolation frame if β=0 is input from the still region detecting portion 102.
Based on two input frames and the control signal from the interpolation vector controlling portion 104, the interpolation frame generating portion 105 generates and outputs an interpolation frame to the subsequent stage based on the interpolation vector from the interpolation vector evaluating portion 103 or zero-vector.
In the case of a video shot while panning a camera at a high speed in a certain direction to follow a subject, for example, in the case of a video shot by following a moving car with a camera as shown by an image example of FIG. 25(A), objects other than a shooting target, i.e., the car, such as buildings and trees in the background portion and a sign passing through the foreside move in the direction opposite to the motion direction of the car at a very high speed in the video. If motion vectors are analyzed in such a video scene, the region of the car becomes an almost still state, and large motion vectors are detected in the direction opposite to the motion direction of the car for the buildings and trees in the background portion and the sign passing through the foreside. That is, when such a video is captured by a computing machine, the region of the car is determined as a low-speed region and the region other than the car is determined as a high-speed region.
In the case of the above video, with regard to the sign passing in front of the car at a high speed, as shown in FIGS. 25(B) and 25(C), a portion of a character may be collapsed to cause color of a portion other than the character to appear inside of the character or cause color of the character to appear on a portion other than the character (portions shown by F of FIG. 25(C)). This collapse is caused by the above zero-vector preferential interpolation process and is likely to occur if the zero-vector interpolation is performed in accordance with the still region determination when an image such as characters having sharp edges at some portions exists within an image having a uniform color like the sign, for example.
FIG. 26 is a diagram for explaining a principle of the collapse generated by executing the zero-vector preferential interpolation process for the still region within the high-speed region. In FIG. 26(A), the zero-vector preferential interpolation process is performed for a region where the FD is diminished, i.e., a portion of β=0, which is defined as a still region in accordance with so-called still region determination. If a peripheral image of the still region moves (scrolls) at a high speed, this causes the collapse since a motion amount of object during one frame period becomes large. In FIG. 26(A), if the interpolation process is executed with zero-vector (arrow shown by a dot line), the background of the character, i.e., an image of the sign itself is interpolated into a region where the character of the sign should normally exist, which is indicated by G within the interpolation frame, and as a result, a portion of the character is lacked.
On the other hand, in the portion of the car, which has a motion vector closer to zero, better image quality can be acquired by executing the zero-vector preferential interpolation process. Describing the reason why better image quality can be acquired by giving priority to zero-vector in the interpolation for the still region within the low-speed region corresponding to the car referring to FIG. 26(B), as shown in a band portion Z, since a composite vector is generated at both ends of the low-speed region from the motion vectors of the high-speed region image (such as buildings, trees, and sign) and the motion vectors of the low-speed region, an interpolation image may be somewhat pulled to the motion direction of the background (high-speed region) image. Therefore, a preferable result is acquired from inside of both ends of the low-speed region by using β=0, i.e., the zero-vector preferential interpolation process, and the zero-vector preferential interpolation process is more appropriate for the still region portion within the low-speed region.
However, even if the zero-vector preferential interpolation process is simply executed for the still region within the low-speed region and the zero-vector preferential interpolation is not executed for the still region within the high-speed region, the collapse may occur in the interpolation image outside of the low-speed region as shown in FIG. 27. For example, if the above interpolation process is applied to an image shown in FIG. 27(A), the collapse may occur in a portion H surrounded by ellipsoids, i.e., both ends of the car corresponding to the ends of the low-speed region as shown in FIG. 27(B). In FIG. 27(B), an image of the low-speed region, i.e., a portion of the car is depicted in a position where the images of the high-speed region (such as buildings and trees) corresponding to the background should normally be depicted.
FIG. 28 is a diagram for explaining a principle of the collapse occurring at outer circumferential ends of the low-speed region. In FIG. 28, if an interpolation process is executed with the use of the motion vector (arrow shown by a dot line), it is problematic that the collapse such as those formed by pulling the low-speed region image occurs due to the effect of the low-speed region image in a region shown by I of the interpolation frame where the high-speed region image should normally exist.
Patent Document 1: Specification of Japanese Patent No. 3295437
Patent Document 2: Japanese Laid-Open Patent Publication No. H6-178285
Non-Patent Document 1: Y. Nojiri, H. Hirabayashi, H. Sonehara, and F. Okano, “HDTV Standards Converter”, Journal of the Institute of Television Engineers of Japan, Vol. 48, No. 1, pp. 84-94 (1994)